<html>
<head><meta charset="utf-8"><title>brainstorming hackmd · t-compiler/shrinkmem-sprint · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/index.html">t-compiler/shrinkmem-sprint</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html">brainstorming hackmd</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="226156390"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/226156390" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#226156390">(Feb 12 2021 at 16:03)</a>:</h4>
<p>Created <a href="https://hackmd.io/Br1k5qy0QWuPdbMZMdyeqQ">hackmd</a> for people to collaboratively post ideas/notes on, as discussed in <a class="stream-topic" data-stream-id="238009" href="/#narrow/stream/238009-t-compiler.2Fmeetings/topic/.5Bplanning.20meeting.5D.202021-02-12">#t-compiler/meetings &gt; [planning meeting] 2021-02-12</a></p>



<a name="226157488"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/226157488" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#226157488">(Feb 12 2021 at 16:11)</a>:</h4>
<p>I can start by adding things from the meeting :)</p>



<a name="226158641"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/226158641" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#226158641">(Feb 12 2021 at 16:19)</a>:</h4>
<p>I also want to mention "send less IR to LLVM" should also help with memory usage, and I've had a vague idea for a while how to do it: <a href="https://github.com/rust-lang/rust/issues/77960">https://github.com/rust-lang/rust/issues/77960</a></p>



<a name="226186436"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/226186436" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cjgillot <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#226186436">(Feb 12 2021 at 19:47)</a>:</h4>
<p>I added a few bullet points to the HackMD.<br>
<span class="user-mention" data-user-id="306073">@Tyson Nottingham</span>: do you have ideas from your investigations of the DepGraph and query system?</p>



<a name="226189813"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/226189813" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#226189813">(Feb 12 2021 at 20:12)</a>:</h4>
<p>wow great start everyone!</p>



<a name="226190281"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/226190281" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tyson Nottingham <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#226190281">(Feb 12 2021 at 20:15)</a>:</h4>
<p><span class="user-mention silent" data-user-id="248906">cjgillot</span> <a href="#narrow/stream/276895-t-compiler.2Fshrinkmem-sprint/topic/brainstorming.20hackmd/near/226186436">said</a>:</p>
<blockquote>
<p><span class="user-mention silent" data-user-id="306073">Tyson Nottingham</span>: do you have ideas from your investigations of the DepGraph and query system?</p>
</blockquote>
<p>Sure. I'll add some notes to the HackMD when I get a chance. I've been looking into the memory usage for a couple months, so I've got a few ideas.</p>



<a name="226190434"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/226190434" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tyson Nottingham <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#226190434">(Feb 12 2021 at 20:16)</a>:</h4>
<p>Mostly outside the DepGraph though tbh</p>



<a name="226196509"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/226196509" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tyson Nottingham <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#226196509">(Feb 12 2021 at 21:09)</a>:</h4>
<p>I'm somewhat actively working on codegen scheduling. There's one low-hanging fruit there that I'm profiling a solution for now (I'll link the PR when it's ready). But there are more gains to be had there, for sure.</p>
<p>I was planning to take on the general scheduling issue, but perhaps someone else should pick it up if there's demand to get it done more quickly. I'm having to scale back working on rustc, as it's become hard to justify it while I'm out of work. I'll create a GH issue / write-up for improving the scheduling more generally and link it here. That should be useful either way.</p>



<a name="226306854"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/226306854" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#226306854">(Feb 14 2021 at 14:28)</a>:</h4>
<blockquote>
<p>Find a way to swap in a custom allocator? (#[global_allocator] won’t work)</p>
</blockquote>
<p>At the risk of asking the obvious, does <code>LD_PRELOAD=/path/to/custommalloc.so</code> not work?</p>



<a name="226340234"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/226340234" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tyson Nottingham <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#226340234">(Feb 15 2021 at 02:51)</a>:</h4>
<p>Here's the PR for the low-hanging codegen scheduling memory reduction I mentioned -- <a href="https://github.com/rust-lang/rust/issues/82127">#82127</a>.</p>



<a name="226399974"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/226399974" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#226399974">(Feb 15 2021 at 15:54)</a>:</h4>
<p><span class="user-mention silent" data-user-id="330154">The 8472</span> <a href="#narrow/stream/276895-t-compiler.2Fshrinkmem-sprint/topic/brainstorming.20hackmd/near/226306854">said</a>:</p>
<blockquote>
<blockquote>
<p>Find a way to swap in a custom allocator? (#[global_allocator] won’t work)</p>
</blockquote>
<p>At the risk of asking the obvious, does <code>LD_PRELOAD=/path/to/custommalloc.so</code> not work?</p>
</blockquote>
<p>Its worth investigating. (The <code>#[global_allocator]</code> wont work comment was in reference to that not intercepting memory requests from LLVM, if I recall correctly. <code>LD_PRELOAD</code> might handle that context fine; I need to look into it.)</p>



<a name="226404092"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/226404092" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#226404092">(Feb 15 2021 at 16:19)</a>:</h4>
<p><span class="user-mention silent" data-user-id="306073">Tyson Nottingham</span> <a href="#narrow/stream/276895-t-compiler.2Fshrinkmem-sprint/topic/brainstorming.20hackmd/near/226340234">said</a>:</p>
<blockquote>
<p>Here's the PR for the low-hanging codegen scheduling memory reduction I mentioned -- <a href="https://github.com/rust-lang/rust/issues/82127">#82127</a>.</p>
</blockquote>
<p>Silly Question: What tools did you use to make those graphs that you posted in the PR? I would like for us all to coalesce around a standard workflow, and its possible your workflow is a good candidate.</p>



<a name="226404438"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/226404438" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#226404438">(Feb 15 2021 at 16:22)</a>:</h4>
<p>(Oh, I see you wrote "Stats were gathered by polling system memory usage once per second via the free command on Linux. The machine was otherwise left alone during benchmarking. I'm sure there's a better way, but this did a passable job." I'd agree: There's probably better, but this is passable, especially since I think everyone will have <code>free</code> installed. Did you just hack together a perl script or something to turn the <code>free</code> outputs into data to feed into gnuplot?)</p>



<a name="226414879"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/226414879" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#226414879">(Feb 15 2021 at 18:03)</a>:</h4>
<p>There also is <code>src/ci/cpu-usage-over-time.py</code>, but that doesn't seem to run on PR builds</p>



<a name="226419451"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/226419451" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#226419451">(Feb 15 2021 at 18:57)</a>:</h4>
<p>there are also fun rust system probes like <a href="https://github.com/kali/readings">https://github.com/kali/readings</a> which does its graphs using the <code>plotters</code> crate IIRC</p>



<a name="226423370"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/226423370" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tyson Nottingham <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#226423370">(Feb 15 2021 at 19:40)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116083">pnkfelix</span> <a href="#narrow/stream/276895-t-compiler.2Fshrinkmem-sprint/topic/brainstorming.20hackmd/near/226404438">said</a>:</p>
<blockquote>
<p>Did you just hack together a perl script or something to turn the <code>free</code> outputs into data to feed into gnuplot?)</p>
</blockquote>
<p>Yeah, just used bash and imported into LibreOffice Calc.</p>
<div class="codehilite" data-code-language="Bash"><pre><span></span><code><span class="nv">time</span><span class="o">=</span><span class="m">0</span><span class="p">;</span> <span class="k">while</span> <span class="o">[</span> -f ../rust/timer.txt <span class="o">]</span><span class="p">;</span> <span class="k">do</span> <span class="nv">mem</span><span class="o">=</span><span class="k">$(</span>free -m <span class="p">|</span> awk <span class="s1">'/Mem/ {print $3}'</span><span class="k">)</span><span class="p">;</span> <span class="nb">echo</span> <span class="s2">"</span><span class="nv">$time</span><span class="s2">,</span><span class="nv">$mem</span><span class="s2">"</span> &gt;&gt; memory_usage.csv<span class="p">;</span> sleep <span class="m">1</span><span class="p">;</span> <span class="nv">time</span><span class="o">=</span><span class="k">$((</span><span class="nb">time</span> <span class="o">+</span> <span class="m">1</span><span class="k">))</span><span class="p">;</span> <span class="k">done</span>
</code></pre></div>



<a name="226460873"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/226460873" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tyson Nottingham <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#226460873">(Feb 16 2021 at 01:08)</a>:</h4>
<p>Added a bunch of ideas related to CGU stuff.</p>



<a name="228297266"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/228297266" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#228297266">(Mar 01 2021 at 20:07)</a>:</h4>
<blockquote>
<p>Might not play well with rustup but should work with rustc?</p>
</blockquote>
<p><code>rustup</code> is a red herring, it's jemalloc that breaks</p>



<a name="228297724"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/228297724" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#228297724">(Mar 01 2021 at 20:10)</a>:</h4>
<p>Anyone want to review <a href="https://hackmd.io/Br1k5qy0QWuPdbMZMdyeqQ">the hackmd</a> with me now and try to identify the ideas that are worthy of publicizing in a dedicated zulip topic in this stream?</p>



<a name="228297903"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/228297903" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cjgillot <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#228297903">(Mar 01 2021 at 20:11)</a>:</h4>
<p>Yep.</p>



<a name="228298362"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/228298362" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#228298362">(Mar 01 2021 at 20:14)</a>:</h4>
<p>lets see.</p>
<h2><a href="https://hackmd.io/Br1k5qy0QWuPdbMZMdyeqQ?view#Tools-for-measuring-memory-usage">tools for measuring memory usage</a></h2>



<a name="228298500"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/228298500" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#228298500">(Mar 01 2021 at 20:15)</a>:</h4>
<p>the section on <code>-Z time-passes</code> leads me to wonder: Is there anyway to make the output reflect the nesting of passes?</p>



<a name="228298542"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/228298542" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#228298542">(Mar 01 2021 at 20:15)</a>:</h4>
<p>not sure. Probably not quickly</p>



<a name="228298756"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/228298756" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#228298756">(Mar 01 2021 at 20:17)</a>:</h4>
<p><span class="user-mention" data-user-id="232545">@Joshua Nelson</span> do you have a recommended strategy for getting around the <code>heaptrack</code> breakage when linking jemalloc statically? E.g., do you turn off jemalloc support in rustc for such builds?</p>



<a name="228298796"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/228298796" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#228298796">(Mar 01 2021 at 20:17)</a>:</h4>
<p><span class="user-mention" data-user-id="116083">@pnkfelix</span> I've never gotten it working. I mainly profile rustdoc so it hasn't bothered me too much</p>



<a name="228298807"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/228298807" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#228298807">(Mar 01 2021 at 20:17)</a>:</h4>
<p>I would expect turning off jemalloc to fix it though</p>



<a name="228298967"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/228298967" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#228298967">(Mar 01 2021 at 20:18)</a>:</h4>
<p>I’ll admit that I’m still confused about our default status with respect to <code>jemalloc</code>. The <code>config.toml.example</code> file leads me to think that we have it off by default, in which case I would not worry about this.</p>



<a name="228299002"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/228299002" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#228299002">(Mar 01 2021 at 20:19)</a>:</h4>
<p>/me goes to see if the jemalloc symbols are in their <code>rustc</code> binary</p>



<a name="228299771"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/228299771" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tyson Nottingham <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#228299771">(Mar 01 2021 at 20:24)</a>:</h4>
<p>It's off by default for local <code>x.py</code> builds, but I'm pretty sure it's enabled for rustc binaries distributed by rustup.</p>



<a name="228301504"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/228301504" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#228301504">(Mar 01 2021 at 20:35)</a>:</h4>
<p>looks like it's set on Mac and Linux at least:</p>
<div class="codehilite"><pre><span></span><code>$ rg jemalloc src/ci
src/ci/github-actions/ci.yml
435:              RUST_CONFIGURE_ARGS: --host=x86_64-apple-darwin --target=x86_64-apple-darwin,aarch64-apple-ios,x86_64-apple-ios --enable-full-tools --enable-sanitizers --enable-profiler --set rust.jemalloc --set llvm.ninja=false
446:              RUST_CONFIGURE_ARGS: --enable-extended --enable-profiler --set rust.jemalloc --set llvm.ninja=false
456:              RUST_CONFIGURE_ARGS: --build=x86_64-apple-darwin --enable-sanitizers --enable-profiler --set rust.jemalloc --set llvm.ninja=false

src/ci/docker/host-x86_64/dist-i686-linux/Dockerfile
97:      --set rust.jemalloc

src/ci/docker/host-x86_64/dist-x86_64-linux/Dockerfile
102:      --set rust.jemalloc
</code></pre></div>



<a name="228301759"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/228301759" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#228301759">(Mar 01 2021 at 20:37)</a>:</h4>
<p>Right, okay; so this is one of those instances where the CI differs from the default developer experience ...</p>



<a name="228312408"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/228312408" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#228312408">(Mar 01 2021 at 21:50)</a>:</h4>
<p>(one of these days I'll put a note about this in the <code>config.toml.example</code>)</p>



<a name="228346222"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/228346222" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tyson Nottingham <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#228346222">(Mar 02 2021 at 03:11)</a>:</h4>
<p>I braindumped a bit about codegen scheduling in <a href="https://github.com/rust-lang/rust/issues/82685">#82685</a>. I've moved on to some other areas of interest, so I'm kind of throwing that over the wall. :)</p>



<a name="228346294"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/228346294" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#228346294">(Mar 02 2021 at 03:12)</a>:</h4>
<p>I pinged a friend who works on scheduling for his PhD thesis, I'll let you know if he's interested :)</p>



<a name="228693889"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/228693889" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#228693889">(Mar 04 2021 at 00:25)</a>:</h4>
<p>Is there any reason why we shouldn't put a default (but user-overridable) upper-bound on the CGU size?</p>



<a name="228694003"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/228694003" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#228694003">(Mar 04 2021 at 00:26)</a>:</h4>
<p>Nothing immediately comes to mind.</p>



<a name="228694008"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/228694008" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#228694008">(Mar 04 2021 at 00:26)</a>:</h4>
<p>I was just musing to myself that the processing of combining them unconditionally until one hits the desired number w.r.t. processor cores is <em>obviously</em> problematic for memory usage</p>



<a name="228694040"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/228694040" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#228694040">(Mar 04 2021 at 00:26)</a>:</h4>
<p>a point that <span class="user-mention" data-user-id="306073">@Tyson Nottingham</span> pointed out in their write-up on <a href="https://github.com/rust-lang/rust/issues/82685">#82685</a>.</p>



<a name="228694081"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/228694081" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#228694081">(Mar 04 2021 at 00:26)</a>:</h4>
<p>but i figured a bound like I outlined should be trivial to implement, and if its large enough, it need not have noticeable impact on most crates...</p>



<a name="228694131"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/228694131" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#228694131">(Mar 04 2021 at 00:27)</a>:</h4>
<p>I was investigating the cgu merging algorithm today and after testing out a few different things, I found no difference in the biggest cgus in <code>rustc_mir</code> because the largest cgus don't appear to be the result of merging smaller ones together. They just start out <em>huge</em>.</p>



<a name="228694717"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/228694717" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#228694717">(Mar 04 2021 at 00:32)</a>:</h4>
<p>So breaking up large cgus before we start the merging process seems worth while to me.</p>



<a name="228696738"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/228696738" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#228696738">(Mar 04 2021 at 00:53)</a>:</h4>
<p>I'd be interested in some data on why they're big - are we including a bunch of generic code? Are they still large with -Zshare-generics (not sure if that precisely does what I want)?</p>
<p>Mostly I'm thinking that if the problem is, loosely, modules that are "too big" that's not great, and I'd like to tackle it in the compiler rather than via education, but it feels like a very different problem to us adding a bunch of code which potentially all gets inlined by LLVM, where splitting may result in worse performance.</p>
<p>My recollection is that our algorithm today for creating modules isn't based on the "usage" graph of function calls etc, but rather is module-based, and then we add a bunch atop that for the used generic code. Maybe that's wrong though?</p>



<a name="228737846"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming%20hackmd/near/228737846" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> tm <a href="https://rust-lang.github.io/zulip_archive/stream/276895-t-compiler/shrinkmem-sprint/topic/brainstorming.20hackmd.html#228737846">(Mar 04 2021 at 09:03)</a>:</h4>
<p>I think that is generally accurate. Modules are primary driving factor behind partitioning, so the generated CGUs tend to have say all HashMaps together, all Vecs together, all Zip iterators together. The call graph is significant for inline functions.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>